Method and apparatus for generating depth information

ABSTRACT

A method and an apparatus for generating depth information are provided. A main depth map associated with one of a left image and a right image and corresponding to multiple first pixels is obtained. The left image or the right image associated with the main depth map is divided into multiple segments according to pixel information, so as to obtain a segment distribution map including the segments. Multiple invalid depth values which do not match to a reliable condition are removed from the main depth map according to the segment distribution map, so as to generate a necessary repair depth map including multiple holes. Multiple optimized depth values are respectively generated for the holes in the necessary repair depth map according to the segment distribution map, and the optimized depth values are filled into the necessary repair depth map to generate an optimized depth map.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the priority benefit of Taiwan applicationserial no. 103114050, filed on Apr. 17, 2014. The entirety of theabove-mentioned patent application is hereby incorporated by referenceherein and made a part of this specification.

FIELD OF THE INVENTION

This invention relates to a method and an apparatus for image processingand more particularly, to a method and an apparatus for generating depthinformation.

DESCRIPTION OF RELATED ART

As image processing technology develops and thrives, stereo vision hasbeen gradually and widely applied to a variety of fields. The stereovision in broad terms can include two stages. In the former stage, depthinformation is generated by using a depth camera, a stereo camera or amethod, such as a three-dimension (3D) imaging algorithm, and in thelater stage, images in different vision angles are generated accordingto the depth information. Accordingly, in order to generate a 3D imagewith better visual perception, accurate depth information plays a veryimportant role.

As for a stereoscopic imaging system having dual lenses, images indifferent angles related to one scene can be captured by using the duallenses of the stereoscopic imaging system, and depth information of eachobject in the images may be estimated according to information of pixeldifference between images captured by the left and the right lenses anddistance between the two lenses. Generally, the depth information can berepresented in a depth map, and depths of different objects or pixels ina two-dimensional (2D) image are represented by using depth valuesrecorded on the depth map. Therein, a depth generation algorithm basedon local-correlation and pixel-wise matching is common to thestereoscopic imaging systems having dual lenses.

However, a depth map generated based on the local-correlation may havean edge blur issue, while the pixel-wise matching method may also leadto significant calculation and great consumption in memory resources. Inother words, different algorithms for generating the depth informationhave different accuracies and calculation amounts. Therefore, how togenerate accurate depth information under a condition with a permissiblecalculation amount and complexity as well as to enhance quality of the3D image generated according to the depth information has been animportant goal to be achieved by the persons skilled in the field.

SUMMARY

Accordingly, the invention provides a method and an apparatus forgenerating depth information capable of reducing noise in the depthinformation and enhancing accuracy of the depth information, such that athree-dimensional (3D) image produced by a stereoscopic imaging systemcan provide a better visual experience.

The invention provides a method for generating depth information for anelectronic apparatus. The method includes the following steps. First, aleft image and a right image captured by a stereoscopic imaging systemare obtained. Three-dimensional (3D) depth estimation is performed onthe left image and the right image so as to obtain a main depth mapassociated with one of the left image and the right image andcorresponding to a plurality of first pixels. The main depth map recordsa plurality of main depth values respectively corresponding to the firstpixels. According to pixel information of each of the first pixels, theleft image or the right image associated with the main depth map isdivided into a plurality of segments, so as to obtain a segmentdistribution map including the segments. Whether the main depth valuescorresponding to the first pixels match a reliable condition isdetermined according to the segment distribution map. A plurality ofinvalid depth values among the main depth values which do not match thereliable condition are removed from the main depth map, so as togenerate the necessary repair depth map including a plurality of thirdholes among the holes. A plurality of optimized depth values aregenerated respectively for the holes in the necessary repair depth mapaccording to the segment distribution map, and the optimized depthvalues are filled into the necessary repair depth map to generate anoptimized depth map.

In an embodiment of the invention, the step of dividing the left imageor the right image associated with the main depth map into the segmentsaccording to the pixel information of each of the first pixels so as toobtain the segment distribution map including the segments includes thefollowing steps. A first adjacent pixel and a second adjacent pixel thatare adjacent to each other among the first pixels are compared todetermine whether a pixel value difference between the first adjacentpixel and the second adjacent pixel is less than a difference threshold.If the pixel value difference is less than the difference threshold, thefirst adjacent pixel and the second adjacent pixel are connected witheach other to form a first segment among the segments. The first segmentat least includes the first adjacent pixel and the second adjacentpixel.

In an embodiment of the invention, the step of dividing the left imageor the right image associated with the main depth map into the segmentsaccording to the pixel information of each of the first pixels so as toobtain the segment distribution map including the segments furtherincludes the following steps. A segment size of each of the segments islimited according to a size threshold to obtain the segment distributionmap. The segment size of each of the segments is not greater than thesize threshold.

In an embodiment of the invention, the step of determining whether themain depth values corresponding to the first pixels match the reliablecondition according to the segment distribution map and removing theinvalid depth values among the main depth values which do not match thereliable condition from the main depth map so as to generate thenecessary repair depth map including multiple holes includes thefollowing steps. The main depth map is divided into a plurality of depthsegments according to the segment distribution map, and a statisticalcalculation is performed on the main depth values in each of the depthsegments to obtain a statistical result. Whether the main depth valuesare a plurality of first invalid depth values among the invalid depthvalues is determined according to the statistical result, and the firstinvalid depth values is removed from the main depth map, so as togenerate the necessary repair depth map including a plurality of firstholes among the holes.

In an embodiment of the invention, after the step of removing the firstinvalid depth values from the main depth map so as to generate thenecessary repair depth map including the first holes among the holes,the method further includes the following steps. A plurality of invaliddensity values of the first invalid depth values within a limited rangeis respectively calculated by using a plurality of first valid depthvalues which are not the first invalid depth values among the main depthvalues as centers. Whether the first valid depth values are a pluralityof second invalid depth values among the invalid depth valuesrespectively is determined according to whether the invalid densityvalues are greater than a density threshold. The second invalid depthvalues are removed from the main depth map, so as to generate thenecessary repair depth map including a plurality of second holes amongthe holes.

According to an embodiment of the present invention, the method furtherincludes the following steps. The 3D depth estimation is performed onthe left image and the right image so as to obtain an auxiliary depthmap associated with the other of the left image and the right image andcorresponding to a plurality of second pixels. The auxiliary depth maprecords an auxiliary depth value of each of the second pixels. Theauxiliary depth map and the main depth map are compared to remove themain depth values which are not consistent with the correspondingauxiliary depth values thereof from the main depth map, so as togenerate the necessary repair depth map including a plurality of thirdholes among the holes.

In an embodiment of the invention, the segment distribution map has asegmentation fineness according to the difference threshold and the sizethreshold. The segment distribution map includes a first segmentdistribution map and a second segment distribution map, and thesegmentation fineness of the first segment distribution map is differentfrom the segmentation fineness of the second segment distribution map.

In an embodiment of the invention, the step of generating the optimizeddepth values respectively for the holes in the necessary repair depthmap according to the segment distribution map and filling the optimizeddepth values into the necessary repair depth map to generate theoptimized depth map includes the following steps. The necessary repairdepth map is divided into a plurality of first depth segments accordingto the first segment distribution map, and a first valid density valueis obtained according to the number of the holes in each of the firstdepth segments. Whether to calculate a first average depth value of eachof the first depth segments is determined according to the first validdensity value, and part of the holes are filled by using the firstaverage depth value as one of the optimized depth values. The precedingtwo steps are repeated until the times of repeating the preceding twosteps reach a predetermined number.

In an embodiment of the invention, the step of determining whether tocalculate the first average depth value of each of the first depthsegments according to the first valid density value includes thefollowing step. Whether to calculate the first average depth value ofeach of the first depth segments is determined according to whether thefirst valid density value is greater than a valid threshold.

In an embodiment of the invention, the step of determining whether tocalculate the first average depth value of each of the first depthsegments according to the first valid density value further includes thefollowing steps. A processing sequence of each of the first depthsegments is obtained according to the first valid density value of eachof the first depth segments, and whether to fill the holes in each ofthe first depth segments is determined according to whether theprocessing sequence has a high priority.

In an embodiment of the invention, after the step of determining whetherto calculate the first average depth value of each of the first depthsegments according to the first valid density value and filling part ofthe holes by using the first average depth value as one of the optimizeddepth values, the method further includes the following steps. Thenecessary repair depth map is divided into a plurality of second depthsegments according to the second segment distribution map, and a secondvalid density value is obtained according to the number of the holes ineach of the second depth segments, whether to calculate a second averagedepth value of each of the second depth segments is determined accordingto the second valid density value, and part of the holes are filled byusing the second average depth value as one of the optimized depthvalues.

In another aspect, the invention provides an apparatus for generatingdepth information, which includes a storage unit recording a pluralityof modules and one or more processing units. The one or more processingunits are coupled with the storage unit to access and execute themodules recorded in the storage unit. The modules include a depthestimation module, a segment distribution map obtaining module, aninvalid depth removing module and a holes filling module. The depthestimation module obtains a left image and a right image captured by astereoscopic imaging system and performs 3D depth estimation on the leftimage and the right image so as to obtain a main depth map associatedwith one of the left image and the right image and corresponding to aplurality of first pixels. The main depth map records a plurality ofmain depth values respectively corresponding to the first pixels. Thesegment distribution map obtaining module divides the left image or theright image associated with the main depth map into a plurality ofsegments according to pixel information of each of the first pixels, soas to obtain a segment distribution map including the segments. Theinvalid depth removing module determines whether the main depth valuescorresponding to the first pixels match a reliable condition accordingto the segment distribution map and removes a plurality of invalid depthvalues among the main depth values which do not match the reliablecondition from the main depth map, so as to generate a necessary repairdepth map including a plurality of holes. The holes filling modulegenerates a plurality of optimized depth values respectively for theholes in the necessary repair depth map according to the segmentdistribution map and fills the optimized depth values into the necessaryrepair depth map to generate an optimized depth map.

To sum up, in the embodiments with respect to the generation of thedepth information of the invention, the original left image or rightimage is divided according to the pixel information to generate thesegment distribution map including the segments. The segmentdistribution map is then used to remove the invalid depth values fromthe depth map. Furthermore, in the invention, the invalid depth valuesin each of the segments can be removed through calculating thestatistical information of the depth values included within each of thesegments. Besides, filling processes of different stages can beperformed by the dividing manners applied to the segments in differentsizes, such that the holes in the depth map can be filled by using theoptimized depth values generated in different stages. Thereby, not onlythe invalid depth values with low reliability can be removed from thedepth map, but also the holes in the depth map can be filled by usingthe optimized depth values generated according to information withrespect to the nearby depth values of each of the holes, so as togenerate the depth map with less noise and high accuracy.

To make the above features and advantages of the invention morecomprehensible, embodiments accompanied with drawings are described indetail below.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are included to provide a furtherunderstanding of the invention, and are incorporated in and constitute apart of this specification. The drawings illustrate embodiments of theinvention and, together with the description, serve to explain theprinciples of the invention.

FIG. 1 is a block diagram illustrating an apparatus for generating depthinformation according to an embodiment of the invention.

FIG. 2 is a flowchart illustrating a method for generating depthinformation according to an embodiment of the invention.

FIG. 3A is a schematic diagram illustrating an example of dividing theleft image or the right image.

FIG. 3B is an exemplary schematic diagram illustrating an example of apart of the segment distribution map.

FIG. 4 is a schematic diagram illustrating the operation of the methodfor generating depth information according to an embodiment of theinvention.

FIG. 5 is a flowchart illustrating a method for generating a necessaryrepair depth map according to an embodiment of the invention.

FIG. 6 is a schematic diagram illustrating an example of removing theinvalid depth values according to a statistical result according to anembodiment of the invention.

FIG. 7 is a schematic diagram illustrating an example of removing theinvalid depth values according to a density according to an embodimentof the invention.

FIG. 8 is a schematic diagram illustrating the operation of filling thenecessary repair depth map according to an embodiment of the invention.

FIG. 9A and FIG. 9B are a flowchart of generating an optimized depth mapaccording to an embodiment of the invention.

FIG. 10 is a schematic diagram illustrating an example of the segmentdistribution map according to the embodiment depicted in FIG. 8.

DESCRIPTION OF EMBODIMENTS

FIG. 1 is a block diagram illustrating an apparatus for generating depthinformation according to an embodiment of the invention. With referenceto FIG. 1, an image processing apparatus 10 of the present embodimentis, for example, a mobile phone, a tablet computer, a notebook computeror a stereoscopic camera including a stereoscopic imaging system (notshown), but the invention is not limited thereto. Namely, the imageprocessing apparatus 10 may be an image capturing apparatus includingthe stereoscopic imaging system. Additionally, the image processingapparatus 10 may also be an electronic apparatus coupled with the imagecapturing apparatus having the stereoscopic imaging system, which is notlimited in the invention. The image processing apparatus 10 includes astorage unit 14 and one or more processing units (only a processing unit16 is illustrated for illustration, but the invention is not limitedthereto), and functions thereof will be as below.

The storage unit 14 is, for example, a random access memory, a flashmemory or any other memory for storing data and a plurality of modules.The modules includes a depth estimation module 142, a segmentdistribution map obtaining module 144, an invalid depth removing module146 and holes filling module 148. The modules are, for example, programswhich may be loaded into the processing unit 16 to execute a functionfor generating depth information. In other words, the processing unit 16is coupled with the storage unit 14 and serves to execute the modules,so as to control the image processing apparatus 10 to execute thefunction for generating the depth information. The processing unit 16may be, for example, a central processing unit (CPU), a microprocessor,an application specific integrated circuits (ASIC), a programmable logicdevice (PLD) or any other hardware device having computing capability.

FIG. 2 is a flowchart illustrating a method for generating depthinformation according to an embodiment of the invention. With referenceto FIG. 2, the method of the present embodiment is suitable for theimage processing apparatus 10 of FIG. 1. Detailed steps of the methodfor generating the depth information operated by using each element ofthe image processing apparatus 10 will be described below.

First, the depth estimation module 142 obtains a left image and a rightimage captured by a stereoscopic imaging system and performsthree-dimensional (3D) depth estimation on the left image and the rightimage, so as to obtain a main depth map associated with one of the leftimage and the right image and corresponding to multiple first pixels(step S201). The main depth map records a plurality of main depth valuesrespectively corresponding to the first pixels. Furthermore, in anembodiment, the stereoscopic imaging system includes two image sensingmodules. The two image sensing modules may classified into a left imagesensing module and a right image sensing module according to positionswhere their lenses are disposed. Accordingly, when a user presses ashutter button or issues a capturing command, the left image sensingmodule and the right image sensing module captures images (i.e., theleft image and the right image) in different angles for the same scene.

In detail, the left image includes a plurality of left pixels, and theright image includes a plurality of right pixels. The depth estimationmodule 142 calculates a pixel difference between each left pixel on theleft image and each right pixel on the right image that arecorresponding to each other. The pixel difference is a displacementbetween the left pixel on the left image and the right pixel on theright image that are corresponding to each other. Accordingly, the depthestimation module 142 may estimates a depth value of each left pixel onthe left image and a depth value of each right pixel on the right imageaccording to focal lengths for the left lens and the right lens tocapture the left image and the right image, distance between the twolenses, and the pixel difference between the corresponding left andright pixels. That is, the depth estimation module 142 may obtain a leftdepth map based on the left image and a right depth map based on theright image.

It should be mentioned that the main depth map of the present embodimentmay be a left depth map based on the left image or the right depth mapbased on the right image, which is not limited in the invention. Namely,when the main depth map is the left depth map based on the left image,the main depth map records a plurality of main depth values respectivelycorresponding to the left pixels. When the main depth map is the rightdepth map based on the right image, the main depth map records aplurality of main depth values respectively corresponding to the rightpixels.

Returning to the process illustrated in FIG. 2, the segment distributionmap obtaining module 144 divides the left image or the right imageassociated with the main depth map into a plurality of segmentsaccording to pixel information of each of the first pixels, so as toobtain a segment distribution map including the segments (step S202).That is, when the main depth map is the left depth map, the segmentdistribution map obtaining module 144 divides the left image into aplurality of segments according to the pixel information of each of theleft pixels, so as to obtain the segment distribution map including thesegments. On the other hand, when the main depth map is the right depthmap, the segment distribution map obtaining module 144 divides the rightimage into a plurality of segments according to the pixel information ofeach of the right pixels, so as to obtain the segment distribution mapincluding the segments.

In an embodiment, the segment distribution map obtaining module 144compares a first adjacent pixel with a second adjacent pixel that areadjacent to each other among the right pixels (or the right pixels) todetermine whether a pixel value difference between the first adjacentpixel and the second adjacent pixel is less than a difference threshold.In this case, the pixel value difference is produced by a pixel value ofthe first adjacent pixel subtracting a pixel value of the secondadjacent pixel. If the pixel value difference is less than thedifference threshold, the segment distribution map obtaining module 144connects the first adjacent pixel with the second adjacent pixel to forma first segment among the segments. The first segment at least includesthe first adjacent pixel and the second adjacent pixel.

In particular, when the main depth map is the left depth map, thesegment distribution map obtaining module 144 may obtain the pixel valueof each of the left pixels by, for example, calculating a colorparameter of each of the left pixels, so as to generate the segmentdistribution map by using the pixel values as the pixel information. Forexample, the segment distribution map obtaining module 144 may obtainthe pixel values of the left pixels in different chrominance channels bycalculating red, green and blue (RGB) chrominance components of the leftpixels. Similarly, the segment distribution map obtaining module 144 mayalso obtain the pixel values of the left pixels in brightness orchrominance channels by calculating a brightness (Y) and a chrominancecomponent (Cb,Cr) of each of the left pixels.

Thereby, the segment distribution map obtaining module 144 may generatethe pixel value difference by comparing pixel values of the adjacentpixels and determine whether to connect the adjacent pixels with eachother according to the pixel value difference between the adjacentpixels. If the pixel value difference is less than the differencethreshold, the adjacent pixels are grouped into the same segment. Inother words, the left image is divided into individual segmentsaccording to color performance of each of the left pixels, and the leftpixels within the same segment have the similar color performance.

For example, FIG. 3A is a schematic diagram illustrating an example ofdividing the left image or the right image. With reference to FIG. 3A,it is assumed that an image Img_1 includes pixels P1 to P5. In thepresent example, the segment distribution map obtaining module 144 firstserves the pixel P1 as a datum point and calculates a pixel value of thepixel P1. Then, the segment distribution map obtaining module 144 firstcompares the pixel value of the pixel P1 with pixel values of the otherpixels therearound. As shown in FIG. 3A, a pixel value differencebetween the pixels P1 and P2 is less than the difference threshold, andthus, the segment distribution map obtaining module 144 connects thepixel P1 with the pixel P2. Similarly, a pixel value difference betweenthe pixels P1 and P4 is less than the difference threshold, and thus,the segment distribution map obtaining module 144 connects the pixel P1with the pixel P4.

Further, a pixel value difference between the pixels P2 and P3 is lessthan the difference threshold, and thus, the segment distribution mapobtaining module 144 connects the pixel P2 with the pixel P3. Incontrast, a pixel value difference between the pixels P3 and P5 isgreater than the difference threshold, and thus, the segmentdistribution map obtaining module 144 does not connect the pixel P3 withthe pixel P5, and the pixel P5 is not grouped into a segment z1. Inbrief, by means of comparing the pixel values of the adjacent pixels,the segment distribution map 144 may obtain the segment z1 formed byusing the pixel P1 as the center datum point, and the color performanceof each pixel within the segment z1 is similar to that of the pixel P1.

Sequentially, according to the position and the pixel value of eachpixel, the image Img_1 is divided into a plurality of segments, and thesegment distribution map obtaining module 144 also obtains a segmentdistribution map including the segments. Accordingly, a segment size ofcoverage of each of the segments is increased with the increase of thedifference threshold. In an embodiment, the segment distribution mapobtaining module 144 may limit the segment size of each of the segmentsaccording to a size threshold to obtain the segment distribution map.Therein, the segment size of each of the segments is not greater thanthe size threshold, and the segment size of coverage each of thesegments is increased with the increase of the difference threshold. Ingeneral, the segment distribution map obtaining module 144 may determinea segmentation fineness of the segment distribution map based on thesettings of the difference threshold and the size threshold.

That is, after determining a value of the difference threshold, a mannerfor setting the center datum point and a value of the size threshold,the segment distribution map obtaining module 144 may obtains a segmentdistribution map with a specific segmentation fineness by dividing theimage into a plurality of segments. For example, FIG. 3B is an exemplaryschematic diagram illustrating an example of a part of the segmentdistribution map. With reference to FIG. 3B, a segment distribution mapm_1 includes a plurality of segments z2 to z6, and areas covered by thesegments z2 to z6 have different sizes and shapes, but the invention isnot limited thereto. Persons with ordinary skills of the art maydetermine the value of the difference threshold, the manner for settingthe center datum point and the value of the size threshold according toactual requirements, which will not be repeatedly described hereinafter.

Back to the process illustrated in FIG. 2, the invalid depth removingmodule 146 determines whether the main depth values corresponding to thefirst pixels match a reliable condition according to the segmentdistribution map and removes a plurality of invalid depth values amongthe main depth values which do not match the reliable condition from themain depth map, so as to generate a necessary repair depth map includinga plurality of holes (step S203). In particular, the invalid depthremoving module 146 may obtain neighboring depth information around eachof the main depth values according to the segment distribution map anddetermines whether each of the main depth values matches the reliablecondition according to the other depth information around each of themain depth values, so as to remove the main depth values which areconsidered as the invalid depth values from the main depth map, so as togenerate the necessary repair depth map.

Then, the holes filling module 148 generates a plurality of optimizeddepth values respectively for the holes in the necessary repair depthmap according to the segment distribution map and fills the optimizeddepth values into the necessary repair depth map to generate anoptimized depth map (step S204). In detail, the segment distribution mapis obtained according to a level of similarity of the pixels and apositional relation among the pixels, and thus, the pixels within thesame segment on the left image or the right image have a certain degreeof relevance. Thereby, the invalid depth removing module 146 may dividethe main depth map generated by the segment distribution map obtainingmodule 144 into a plurality of depth segments according to the segmentdistribution map, and the main depth values within the same depthsegment should also have a certain degree of relevance.

Accordingly, based on the characteristic that the main depth valueswithin the same depth segment have a certain degree of relevance, theinvalid depth removing module 146 may determine whether each of the maindepth values is the invalid depth value, and the holes filling module148 fills the holes generated due to the invalid depth values beingremoved to generate better optimized depth values. Thereby, the imageprocessing apparatus 10 of the present embodiment may further performthe optimization operation of the depth map by using the image contentprovided by the original left image or right image, so as to generate anoptimized depth map with a higher accuracy.

In order to describe the invention more clearly, FIG. 4 is a schematicdiagram illustrating the operation of the method for generating depthinformation according to an embodiment of the invention. With referenceto FIG. 4, the depth estimation module 142 obtains a left image Img_Land a right image Img_R captured by a stereoscopic imaging system andperforms 3D depth estimation on the left image Img_L and the right imageImg_R, so as to obtain a main depth map dm_1 and an auxiliary depth mapdm_2. In the present embodiment, the main depth map dm_1 is, forexample, a left depth map based on the left image Img_L, and theauxiliary depth map dm_2 is, for example, a right depth map based on theright image Img_R.

Accordingly, the segment distribution map obtaining module 144 dividesthe left image Img_L into a plurality of segments according to pixelinformation of the left image Img_L, so as to obtain a segmentdistribution map m_2 including the segments. The invalid depth removingmodule 146 digs holes on the main depth map dm_1 according to thesegment distribution map m_2 and the auxiliary depth map dm_2, so as togenerate a necessary repair depth map dm_3 including a plurality ofholes. The holes filling module 148 generates a plurality of optimizeddepth values respectively for the holes in the necessary repair depthmap dm_3 according to the segment distribution map m_2 and fills theoptimized depth values into the necessary repair depth map dm_3 togenerate an optimized depth map dm_4.

According to the preceding embodiments, the segment distribution mapobtaining module 144 may obtain the segment distribution map m_2 havingdifferent segmenting manners according to different differencethresholds, size thresholds and manners for setting the datum point.Details with respect to the segment distribution map obtaining module144 generating the segment distribution map m_2 has been described aboveand thus, will not be repeated hereinafter. Embodiments will providedbelow to describe in detail how the invalid depth removing module 146and the holes filling module 148 remove the invalid depth values andgenerate the optimized depth values by using the segment distributionmap m_2. FIG. 5 is a flowchart illustrating a method for generating anecessary repair depth map according to an embodiment of the invention,and detailed steps of the method for generating the necessary repairdepth map of the present embodiment will be described with reference toFIG. 4.

Referring to both FIG. 4 and FIG. 5, the invalid depth removing module146 divides the main depth map dm_1 into a plurality of depth segmentsaccording to the segment distribution map m_2 and performs a statisticalcalculation on the main depth values in each of the depth segments toobtain a statistical result (step S501). The statistical calculation maybe, for example, an average calculation, a mode calculation, or a mediancalculation, which is not limited in the invention. For instance, theinvalid depth removing module 146 may perform the average calculation onthe main depth values in each of the depth segments to obtain an averagedepth value of each of the depth segments. Nevertheless, in otherfeasible embodiments, persons with ordinary skills of the art may selectanother appropriate statistical calculation method according actualdemands, so as to determine whether the main depth values are validdepth values according to an appropriate statistical result, which isnot repeatedly described hereinafter.

Then, the invalid depth removing module 146 determines whether the maindepth values are a plurality of first invalid depth values among theinvalid depth values according to the statistical result and removes thefirst invalid depth values from the main depth map dm_1, so as togenerate the necessary repair depth map including a plurality of firstholes among the holes (step S502). Namely, the invalid depth removingmodule 146 may determine whether the main depth values in each of themain depth segments include the first invalid depth values according tothe statistical result and each of the first invalid depth values may beconsidered as invalid if having a great difference from the other maindepth values within the same segment. Accordingly, the invalid depthremoving module 146 removes the first invalid depth values from the maindepth map dm_1 to generate first holes corresponding to the firstinvalid depth values.

For instance, FIG. 6 is a schematic diagram illustrating an example ofremoving the invalid depth values according to a statistical resultaccording to an embodiment of the invention. With reference to FIG. 6,it is assumed that after being divided according to the segmentdistribution map m_2, the main depth map dm_1 may include a depthsegment dz_1 and a depth segment dz_2. In the present example, theinvalid depth removing module 146 may perform the average calculation onmain depth values in the depth segment dz_1 to obtain an average depthvalue of the depth segment dz_1. Thus, the invalid depth removing module146 may determine whether the main depth values in the depth segmentdz_1 are the first invalid depth values according to the average depthvalue. In the present example, a main depth value d1 of the depthsegment dz_1 has a significant difference from the average depth valueof the depth segment dz_1, and thus, the main depth value d1 isdetermined as one first invalid depth value.

Similarly, main depth values d2 to d3 in the depth segment dz_1 aresignificantly different from the average depth value of the depthsegment dz_1, and thus, the main depth values d2 to d3 are alsodetermined as the first invalid depth values. Accordingly, the invaliddepth removing module 146 removes the main depth value d1 considered asone first invalid depth value from the main depth map dm_1, so as togenerate the necessary repair depth map dm_3 including a hole h1.Similarly, the invalid depth removing module 146 removes the main depthvalues d2 to d3 considered as the first invalid depth value from themain depth map dm_1, so as to generate the necessary repair depth mapdm_3 respectively including holes h2 and h3.

It should be mentioned that after the processing of steps S501 and S502,a plurality of holes are generated on the necessary repair depth mapdm_3 due to the first invalid depth values being removed. Accordingly,in the present embodiment, the invalid depth removing module 146 maydetermine a reliability degree of the main depth values according to adensity of the invalid depth values around the unremoved valid depthvalues or a density of the valid depth values.

Thus, the invalid depth removing module 146 calculates a plurality ofinvalid density values of the first invalid depth values within alimited range respectively by using a plurality of first valid depthvalues which are not the first invalid depth values among the main depthvalues as centers (step S503). A size of the limited range may bedetermined according to actual application situations, which is notlimited in the invention. For example, the size of the limited range maybe 5×5 pixels, 10×10 pixels or the like.

Then the invalid depth removing module 146 determines whether the firstvalid depth values are respectively a plurality of second invalid depthvalues among the invalid depth values according to whether the invaliddensity values are greater than a density threshold (step S504). To bemore specific, if the number of the valid depth values around a depthvalue is too small, the reliability degree will be reduced relatively.Thus, the invalid depth removing module 146 may determine whether theunremoved first valid depth values are the second invalid depth valuesaccording to the distribution of the valid depth values. Then, theinvalid depth removing module 146 removes the second invalid depthvalues from the main depth map dm_1, so as to generate the necessaryrepair depth map including a plurality of second holes among the holes(step S505). Therein, persons with ordinary skills of the art may designthe density threshold according to actual demands, and the invention isnot intended to limit the density threshold.

For instance, FIG. 7 is a schematic diagram illustrating an example ofremoving the invalid depth values according to a density according to anembodiment of the invention. With reference to FIG. 7, in the presentexample, the main depth map dm_1 includes a depth value dz_4, and themain depth value dz_4 is an unremoved valid depth value. The invaliddepth removing module 146 uses the main depth value dz_4 as a center tocalculate the number of the invalid depth values within a limited rangeR1, and the invalid depth removing module 146 calculates an invaliddensity value associated with the main depth value dz_4 according to thenumber of the invalid depth values. Referring to the example illustratedin FIG. 7, there are 17 invalid depth values (presented by usingslashes) within the limited range R1 having a size of 5×5 pixels, whichmeans that the invalid density value associated with the main depthvalue dz_4 is high. In other words, the reliability degree of the maindepth value dz_4 is low. Accordingly, in the example illustrated in FIG.7, the invalid depth removing module 146 removes the main depth valuedz_4 considered as one second invalid depth value from the main depthmap dm_1, so as to generate the necessary repair depth map dm_3including a hole h4.

Back to the process illustrated in FIG. 5, the invalid depth removingmodule 146 compares the auxiliary depth map dm_2 with the main depth mapdm_1 to remove the main depth values which is not consistent with thecorresponding auxiliary depth values thereof from the main depth mapdmi, so as to generate the necessary repair depth map dm_3 including aplurality of third holes among the holes (step S506). In particular, theinvalid depth removing module 146 may determine the reliability degreeof the depth values by cross-comparing the left depth map with the rightdepth map. If one main depth value on the main depth map dm_1 isinconsistent with the corresponding auxiliary depth value, the maindepth value is considered as an invalid depth value.

Generally speaking, in the present embodiment, a first-stage process ofremoving the invalid depth values is performed on the main depth map byusing the segment distribution map and the statistical calculation.Then, a second-stage process of removing the invalid depth values isperformed according to the density of the valid depth values. Lastly, athird-stage process of removing the invalid depth values is performed bycross-comparing the main depth map with the auxiliary depth map. Afterthe three stages of processes of removing the invalid depth values, aplurality of invalid depth values with low reliability can be removedfrom the depth map. After the invalid depth values are removed togenerate the necessary repair depth map including the holes, optimizeddepth values close to a real situation may be generated according to thesegment distribution map, so as to generate an optimized depth in thesame way in the invention. An embodiment is illustrated below todescribe details with respect to a holes filling operation of the holesfilling module of the invention.

FIG. 8 is a schematic diagram illustrating the operation of filling thenecessary repair depth map according to an embodiment of the invention.With reference to FIG. 8, the holes filling module 148 fills thenecessary repair depth map dm_3 according to the segment distributionmap generated by the segment distribution map obtaining module 144, soas to generate the optimized depth map dm_4. It should be mentioned thatreferring to the example illustrated in FIG. 8, the segment distributionmap generated by the segment distribution map obtaining module 144includes a first segment distribution map group m_f and a second segmentdistribution map group m_c, and a segmentation fineness corresponding tothe first segment distribution map group m_f is different from asegmentation fineness corresponding to the second segment distributionmap group m_c.

In brief, based on the descriptions related to FIG. 2 and FIG. 3, thesegmentation fineness of the segment distribution map may be determinedaccording to the difference threshold and the size threshold. That is,the segment distribution map obtaining module 144 may generate the firstsegment distribution map group m_f and the second segment distributionmap group m_c which have different segmentation fineness according tothe settings of the difference threshold and the size threshold. Thefirst segment distribution map group m_f includes a plurality of firstsegment distribution maps, and the second segment distribution map groupm_c includes a plurality of second segment distribution maps.

Furthermore, in a scenario where the first segment distribution mapgroup m_f has a fixed segmentation fineness based on a set of adifference threshold and a size threshold, the segment distribution mapobtaining module 144 may generate a plurality of first segmentdistribution maps within the first segment distribution map group m_faccording to different manners for setting a center datum point. Forexample, the first segment distribution map group m_f includes firstsegment distribution maps m_f1 and m_f2. The first segment distributionmaps m_f1 and m_f2 correspond to the same segmentation fineness, buthave different manners for dividing segments.

Similarly, in a scenario where the second segment distribution map groupm_c has a fixed segmentation fineness based on a set of a differencethreshold and a size threshold, the segment distribution map obtainingmodule 144 may generate a plurality of first segment distribution mapswithin the second segment distribution map group m_c according todifferent manners for setting a center datum point. For example, thesecond segment distribution map group m_f includes second segmentdistribution maps m_c1 and m_c2. The second segment distribution mapsm_c1 and m_c2 correspond to the same segmentation fineness, but havedifferent manners for dividing the segments. Accordingly, in anembodiment, the holes filling module 148 may fill the holes on thenecessary repair depth map dm_3 according to the segment distributionmaps corresponding to different segmentation fineness. On the otherhand, the holes filling module 148 may also fill the holes on thenecessary repair depth map dm_3 according to the segment distributionmaps having different manners for dividing the segments.

FIG. 9A and FIG. 9B are a flowchart of generating an optimized depth mapaccording to an embodiment of the invention, and detailed steps forgenerating the optimized depth map of the present embodiment will bedescribed with reference to FIG. 8, FIG. 9A and FIG. 9B. It should bementioned that in the present embodiment, the holes filling module 148performs three stages of processes for filling the holes for thenecessary repair depth map dm_3. In a first-stage holes filling process,the holes filling module 148 fills the holes by using the first segmentdistribution map group m_f having a denser dividing manner. In asecond-stage holes filling process, the holes filling module 148 fillsthe holes by using the second segment distribution map group m_c havinga rougher dividing manner. Namely, in the present exemplary embodiment,the segmentation fineness of the first segment distribution map groupm_f is denser than the segmentation fineness of the second segmentdistribution map group m_c.

Lastly, in a third-stage holes filling process, the holes filling module148 fills the holes by using a third segment distribution map group. Itis to be mentioned that a segmentation fineness of the third segmentdistribution map group is even denser than the segmentation fineness ofthe second segment distribution map group. It is to be mentioned that inan embodiment, the first segment distribution map group m_f may bedirectly used as the third segment distribution map group in the thirdstage, but the invention is not limited thereto. In another embodiment,the segmentation fineness corresponding to the third segmentdistribution map group may be different from the segmentation finenesscorresponding to the first segment distribution map group m_f.

With reference to FIG. 8, FIG. 9A and FIG. 9B, the holes filling module148 divides the necessary repair depth map dm_3 into a plurality offirst depth segments according to the first segment distribution mapm_f1 and obtains a first valid density value according to the number ofthe holes in each of the first depth segments (step S901). Accordingly,the depth values within the same first depth segment have a certaindegree of relevance, and each depth value within the same first depthsegment should be approximate to one another. Thus, the holes fillingmodule 148 determines whether to calculate a first average depth valueof each of the first depth segments according to the first valid densityvalue, and fills part of the holes by using the first average depthvalue as one of the optimized depth values. Namely, in the embodimentsof the invention, the holes filling module 148 further determineswhether the valid depth values are sufficient in each of the first depthsegments according to the valid density values of the first depthsegments. In a scenario where the valid depth values are sufficient, theholes filling module 148 may generate the optimized depth values withhigh reliability.

To be more detailed, step S902 may be divided into sub steps S9021through S9023. First, the holes filling module 148 determines whether tocalculate the first average depth value of each of the first depthsegments according to whether the first valid density value is greaterthan a valid threshold (step S9021). In other words, the holes fillingmodule 148 first filters the first depth segments with low reliability.Then, the holes filling module 148 obtains a processing sequence of eachof the first depth segments according to the first valid density valueof each of the first depth segments and determines whether to fill theholes in each of the first depth segments according to whether theprocessing sequence has a high priority (step S9021).

In particular, the holes filling module 148 may calculate the validdensity value of each of the first depth segments and sort each of thefirst depth segments according to high and low of each valid densityvalues. Accordingly, the holes filling module 148 may learn which firstdepth segments have higher degrees of reliability, so as to generate theoptimized depth values based on sufficient valid depth values. Forexample, after sorting each of the first depth segments according tohigh and low of each valid density values, the holes filling module 148first selects the first depth segments having the higher valid depthvalues according to a predetermined proportion threshold. Thepredetermined proportion threshold may fall within a range from 10% to70%, for example, which construes no limitations to the invention.Persons with ordinary skills of the art may determine the proportionthreshold depending on actual application situations, which is notlimited in the invention. After which first depth segments are selectedfor filling the holes therein, the holes filling module 148 calculates afirst average depth value for the selected first depth segments andfills part of the holes by using the first average depth value as one ofthe optimized depth values (step S9023).

Accordingly, in the present embodiment, the holes filling module 148does not fill all the holes in one time, but fills the holes in the areawith high reliability first. Therefore, the holes filling module 148repeats steps S901 and S902 until the times of repeating the precedingtwo steps reach a predetermined number (step S903). The predeterminednumber of times may fall within a range from 10 to 100 times, forexample, which construes no limitations to the invention. Persons withordinary skills of the art may determine the number of times dependingon actual application situations, which is not limited in the invention.

It should be mentioned that during the process of repeating steps S901and S902, the holes filling module 148 may also fill the holes accordingto the plurality of first segment distribution maps having differentmanners for dividing the segments in the first segment distribution mapgroup m_f. For instance, the holes filling module 148 may fill the holesfor the necessary repair depth map dm_3 by sequentially using the firstsegment distribution maps m_f1 and m_f2, where the first segmentdistribution maps m_f1 and m_f2 have the same segmentation fineness.

After the first-stage process for filling the holes is completed, theholes filling module 148 fills the holes by using the second segmentdistribution map group m_c having a rougher segmentation fineness. Forinstance, FIG. 10 is a schematic diagram illustrating an example of thesegment distribution map according to the embodiment depicted in FIG. 8.With reference to FIG. 10, the first segment distribution maps m_f1 andm_f2 of the first segment distribution map group m_f have differentmanners for dividing the depth segments according to different mannersfor setting their center datum points, but have the segmentationfineness. In other words, segments dz_a and dz_b have approximatesegment sizes, but different segment shapes.

In the same way, the second segment distribution maps m_c1 and m_c2 ofthe second segment distribution map group m_c have different manners fordividing the depth segments according to different manners for settingtheir center datum points, but have the segmentation fineness. Thesegmentation fineness of the first segment distribution map group m_f isdenser than the segmentation fineness of the second segment distributionmap group m_c. Referring to FIG. 10, the segment size of the relevantsegment dz_a on the first segment distribution map m_f1 is smaller thanthe segment size of the relevant segment dz_b on the second segmentdistribution map m_c1.

Back to the process illustrated in FIG. 9, the holes filling module 148divides the necessary repair depth map dm_3 into a plurality of seconddepth segments according to the second segment distribution map m_c1 andobtains a second valid density value according to the number of theholes in each of the second depth segments (step S904). Similarly, theholes filling module 148 determines whether to calculate a secondaverage depth value of each of the second depth segments according tothe second valid density value and fills part of the holes by using thesecond average depth value as one of the optimized depth values (stepS905). It should be mentioned that the second segment distribution mapgroup m_c with a rougher segmentation fineness may be used to fill partof the holes in the necessary repair depth map dm_3 which have uncleartextures and weak contour outline information.

Furthermore, step S905 may be divided into sub steps S9051 throughS9052. The holes filling module 148 obtains a processing sequence ofeach of the second depth segments according to the second valid densityvalue of each of the second depth segments and determines whether tofill the holes in each of the second depth segments according to whetherthe processing sequence has a high priority (step S9051). The holesfilling module 148 calculates the second average depth value and fillspart of the holes by using the second average depth value as one of theoptimized depth values (step S9052). The holes filling module 148repeats steps S904 and S905 until the times of repeating steps S904 andS905 reaches a predetermined number (step S906). Details with respect tothe operation of the holes filling module 148 filling the holesaccording to the second segment distribution map group m_c is similar tothat of filling the holes according to the first segment distributionmap group m_f, which can be deduced by persons with ordinary skills ofthe art based on the above descriptions, and will not be repeated anylonger.

After the second-stage process for filling the holes is completed, theholes filling module 148 divides the necessary repair depth map dm_3into a plurality of third depth segments according to the third segmentdistribution map and obtains a third valid density value according tothe number of the holes in each of the third depth segments (step S907).The holes filling module 148 determines whether to calculate a thirdaverage depth value of each of the third depth segments according to thethird valid density value and fills part of the holes by using the thirdaverage depth value as one of the optimized depth values (step S908).

Sequentially, step S908 may be divided into sub steps S9081 throughS9082. The holes filling module 148 obtains a processing sequence ofeach of the third depth segments according to the third valid densityvalue of each of the third depth segments and determines whether to fillthe holes in each of the third depth segments according to whether theprocessing sequence has a high priority (step S9081). The holes fillingmodule 148 calculates the third average depth value and fills part ofthe holes by using the third average depth value as one of the optimizeddepth values (step S9082).

The holes filling module 148 repeats steps S907 and S908 until the timesof repeating steps S907 and S908 reach a predetermined number (stepS909). Details with respect to the operation of the holes filling module148 filling the holes according to the third segment distribution mapgroup m_c is similar to that of filling the holes according to the firstsegment distribution map group m_f, which can be deduced by persons withordinary skills of the art based on the above descriptions, and will notbe repeated any longer. Accordingly, through the three stages ofprocesses for filling the holes, the holes filling module 148 may fillthe holes on the necessary repair depth map dm_3 sequentially accordingto the segment distribution maps corresponding to different segmentationfineness, so as to generate an optimized depth map with a high accuracy.

It is to be mentioned that the depth map generated by performing the 3Ddepth estimation on the left image and the right image generally havepartial unknown areas, e.g., unknown areas may be generated in a leftedge or a right edge of the depth map due to insufficient informationfor generating the depth values. Since the periphery of the unknownareas has not valid depth values, only wrong values can be gotten nomatter how to get values from neighboring areas for filling the holes.However, in the invention, the holes are filled according to the segmentdistribution map and specific filling conditions. Thus, the holes in theareas with the higher reliability may be first filled by using thesegment distribution map, so as to generate a good optimized depth mapin a way gradually spread from the high reliability. Accordingly, theoptimized depth map of the invention leads the unknown areas to a goodoptimization effect.

To sum up, in the embodiments with respect to the generation of thedepth information of the invention, the depth map optimization may beperformed on the depth map based on the information provided by the leftimage and the right image of the original image. Furthermore, in theinvention, the original left image or right image is first dividedaccording to the pixel information to generate a segment distributionmap including plurality of segments, and the segment distribution map isthen used to remove the invalid depth values from the depth map.Thereafter, the holes can be filled by the processes of different stagesaccording to the dividing manners applied to the segments in differentsizes, such that the holes in the depth map can be filled by using theoptimized depth values generated in different stages. Thereby, not onlythe invalid depth values with low reliability can be removed from thedepth map, but also the holes in the depth map can be filled by usingthe optimized depth values generated according to information withrespect to the periphery of the holes, so as to generate the depth mapwith less noise and high accuracy.

Although the invention has been disclosed by the above embodiments, theyare not intended to limit the invention. It will be apparent to one ofordinary skill in the art that modifications and variations to theinvention may be made without departing from the spirit and scope of theinvention. Therefore, the scope of the invention will be defined by theappended claims.

What is claimed is:
 1. A method for generating depth information, for anelectronic apparatus, the method comprising: obtaining a left image anda right image captured by a stereoscopic imaging system and performingthree-dimensional (3D) depth estimation on the left image and the rightimage so as to obtain a main depth map associated with one of the leftimage and the right image and corresponding to a plurality of firstpixels, wherein the main depth map records a plurality of main depthvalues respectively corresponding to the first pixels; dividing the leftimage or the right image associated with the main depth map into aplurality of segments according to pixel information of each of thefirst pixels, so as to obtain a segment distribution map including thesegments; determining whether the main depth values corresponding to thefirst pixels match a reliable condition according to the segmentdistribution map and removing a plurality of invalid depth values amongthe main depth values which do not match the reliable condition from themain depth map, so as to generate a necessary repair depth map includinga plurality of holes, comprising: dividing the main depth map into aplurality of depth segments according to the segment distribution mapand performing a statistical calculation on the main depth values ineach of the depth segments to obtain a statistical result; anddetermining whether the main depth values are a plurality of firstinvalid depth values among the invalid depth values according to thestatistical result and removing the first invalid depth values from themain depth map, so as to generate the necessary repair depth mapincluding a plurality of first holes among the holes; and generating aplurality of optimized depth values respectively for the holes in thenecessary repair depth map according to the segment distribution map andfilling the optimized depth values into the necessary repair depth mapto generate an optimized depth map.
 2. The method according to claim 1,wherein the step of dividing the left image or the right imageassociated with the main depth map into the segments according to thepixel information of each of the first pixels so as to obtain thesegment distribution map including the segments comprises: comparing afirst adjacent pixel with a second adjacent pixel that are adjacent toeach other among the first pixels to determine whether a pixel valuedifference between the first adjacent pixel and the second adjacentpixel is less than a difference threshold; and if the pixel valuedifference is less than the difference threshold, connecting the firstadjacent pixel with the second adjacent pixel to form a first segmentamong the segments, wherein the first segment comprises the firstadjacent pixel and the second adjacent pixel.
 3. The method according toclaim 2, wherein the step of dividing the left image or the right imageassociated with the main depth map into the segments according to thepixel information of each of the first pixels so as to obtain thesegment distribution map including the segments comprises: limiting asegment size of each of the segments according to a size threshold toobtain the segment distribution map, wherein the segment size of each ofthe segments is not greater than the size threshold.
 4. The methodaccording to claim 1, wherein after the step of removing the firstinvalid depth values from the main depth map so as to generate thenecessary repair depth map including the first holes among the holes,the method further comprises: calculating a plurality of invalid densityvalues of the first invalid depth values within a limited rangerespectively by using a plurality of first valid depth values which arenot the first invalid depth values among the main depth values ascenters; determining whether the first valid depth values are aplurality of second invalid depth values among the invalid depth valuesrespectively according to whether the invalid density values are greaterthan a density threshold; and removing the second invalid depth valuesfrom the main depth map, so as to generate the necessary repair depthmap including a plurality of second holes among the holes.
 5. The methodaccording to claim 1, further comprising: performing the 3D depthestimation on the left image and the right image so as to obtain anauxiliary depth map associated with the other of the left image and theright image and corresponding to a plurality of second pixels, whereinthe auxiliary depth map records an auxiliary depth value of each of thesecond pixels; and comparing the main depth map with the auxiliary depthmap to remove the main depth values which is not consistent with thecorresponding auxiliary depth values thereof from the main depth map, soas to generate the necessary repair depth map including a plurality ofthird holes among the holes.
 6. The method according to claim 3, whereinthe segment distribution map has a segmentation fineness according tothe difference threshold and the size threshold, the segmentdistribution map comprises a first segment distribution map and a secondsegment distribution map, and the segmentation fineness of the firstsegment distribution map is different from the segmentation fineness ofthe second segment distribution map.
 7. The method according to claim 6,wherein the step of generating the optimized depth values respectivelyfor the holes in the necessary repair depth map according to the segmentdistribution map and filling the optimized depth values into thenecessary repair depth map to generate the optimized depth mapcomprises: dividing the necessary repair depth map into a plurality offirst depth segments according to the first segment distribution map andobtaining a first valid density value according to the number of theholes in each of the first depth segments; determining whether tocalculate a first average depth value of each of the first depthsegments according to the first valid density value and filling part ofthe holes by using the first average depth value as one of the optimizeddepth values; and repeating the preceding two steps until the times ofrepeating the preceding two steps reach a predetermined number.
 8. Themethod according to claim 7, wherein the step of determining whether tocalculate the first average depth value of each of the first depthsegments according to the first valid density value comprises:determining whether to calculate the first average depth value of eachof the first depth segments according to whether the first valid densityvalue is greater than a valid threshold.
 9. The method according toclaim 7, wherein the step of determining whether to calculate the firstaverage depth value of each of the first depth segments according to thefirst valid density value further comprises: obtaining a processingsequence of each of the first depth segments according to the firstvalid density value of each of the first depth segments and determiningwhether to fill the holes in each of the first depth segments accordingto whether the processing sequence has a high priority.
 10. The methodaccording to claim 7, wherein after the step of determining whether tocalculate the first average depth value of each of the first depthsegments according to the first valid density value and filling part ofthe holes by using the first average depth value as one of the optimizeddepth values, the method further comprises: dividing the necessaryrepair depth map into a plurality of second depth segments according tothe second segment distribution map and obtaining a second valid densityvalue according to the number of the holes in each of the second depthsegments; and determining whether to calculate a second average depthvalue of each of the second depth segments according to the second validdensity value and filling part of the holes by using the second averagedepth value as one of the optimized depth values.
 11. An apparatus forgenerating depth information, comprising: a storage unit, recording aplurality of modules; and one or more processing units, coupled with thestorage unit to access and execute the modules recorded in the storageunit, wherein the modules comprise: a depth estimation module, obtaininga left image and a right image captured by a stereoscopic imaging systemand performing 3D depth estimation on the left image and the right imageso as to obtain a main depth map associated with one of the left imageand the right image and corresponding to a plurality of first pixels,wherein the main depth map records a plurality of main depth valuesrespectively corresponding to the first pixels; a segment distributionmap obtaining module, dividing the left image or the right imageassociated with the main depth map into a plurality of segmentsaccording to pixel information of each of the first pixels, so as toobtain a segment distribution map including the segments; an invaliddepth removing module, determining whether the main depth valuescorresponding to the first pixels match a reliable condition accordingto the segment distribution map and removing a plurality of invaliddepth values among the main depth values which do not match the reliablecondition from the main depth map, so as to generate a necessary repairdepth map including a plurality of holes, wherein the invalid depthremoving module divides the main depth map into a plurality of depthsegments according to the segment distribution map, performs astatistical calculation on the main depth values in each of the depthsegments to obtain a statistical result, determines whether the maindepth values are a plurality of first invalid depth values among theinvalid depth values according to the statistical result and removes thefirst invalid depth values from the main depth map, so as to generatethe necessary repair depth map including a plurality of first holesamong the holes; and a holes filling module, generating a plurality ofoptimized depth values respectively for the holes in the necessaryrepair depth map according to the segment distribution map and fillingthe optimized depth values into the necessary repair depth map togenerate an optimized depth map.
 12. The apparatus according to claim11, wherein the segment distribution map obtaining module compares afirst adjacent pixel with a second adjacent pixel that are adjacent toeach other among the first pixels to determine whether a pixel valuedifference between the first adjacent pixel and the second adjacentpixel is less than a difference threshold, if the pixel value differenceis less than the difference threshold, the segment distribution mapobtaining module connects the first adjacent pixel with the secondadjacent pixel to form a first segment among the segments, wherein thefirst segment comprises the first adjacent pixel and the second adjacentpixel.
 13. The apparatus according to claim 12, wherein the segmentdistribution map obtaining module limits a segment size of each of thesegments according to a size threshold to obtain the segmentdistribution map, and the segment size of each of the segments is notgreater than the size threshold.
 14. The apparatus according to claim11, wherein the invalid depth removing module further calculates aplurality of invalid density values of the first invalid depth valueswithin a limited range respectively by using a plurality of first validdepth values which are not the first invalid depth values among the maindepth values as centers, determines whether the first valid depth valuesare a plurality of second invalid depth values among the invalid depthvalues respectively according to whether the invalid density values aregreater than a density threshold and removes the second invalid depthvalues from the main depth map, so as to generate the necessary repairdepth map including a plurality of second holes among the holes.
 15. Theapparatus according to claim 11, wherein the depth estimation modulefurther performs the 3D depth estimation on the left image and the rightimage so as to obtain an auxiliary depth map associated with the otherof the left image and the right image and corresponding to a pluralityof second pixels, wherein the auxiliary depth map records an auxiliarydepth value of each of the second pixels, wherein the invalid depthremoving module compares the main depth map with the auxiliary depth mapto remove the main depth values which is not consistent with thecorresponding auxiliary depth values thereof from the main depth map, soas to generate the necessary repair depth map including a plurality ofthird holes among the holes.
 16. The apparatus according to claim 13,wherein the segment distribution map has a segmentation finenessaccording to the difference threshold and the size threshold, thesegment distribution map comprises a first segment distribution map anda second segment distribution map, and the segmentation fineness of thefirst segment distribution map is different from the segmentationfineness of the second segment distribution map.
 17. The apparatusaccording to claim 16, wherein the holes filling module divides thenecessary repair depth map into a plurality of first depth segmentsaccording to the first segment distribution map, obtains a first validdensity value according to the number of the holes in each of the firstdepth segments, determines whether to calculate a first average depthvalue of each of the first depth segments according to the first validdensity value and fills part of the holes by using the first averagedepth value as one of the optimized depth values.
 18. The apparatusaccording to claim 17, wherein the holes filling module determineswhether to calculate the first average depth value of each of the firstdepth segments according to whether the first valid density value isgreater than a valid threshold.
 19. The apparatus according to claim 17,wherein the holes filling module obtains a processing sequence of eachof the first depth segments according to the first valid density valueof each of the first depth segments and determining whether to fill theholes in each of the first depth segments according to whether theprocessing sequence has a high priority.
 20. The apparatus according toclaim 17, wherein the holes filling module divides the necessary repairdepth map into a plurality of second depth segments according to thesecond segment distribution map, obtains a second valid density valueaccording to the number of the holes in the each of the second depthsegments, determines whether to calculate a second average depth valueof each of the second depth segments according to the second validdensity value and fills part of the holes by using the second averagedepth value as one of the optimized depth values.